ADFA-5153: Migration script for the shared Brotli dictionary - #1724

…tation.db

Recompresses every brotli Content row against the dictionary already in the
database's CompressionDictionary table. Written for the 20-Aug database, which
has the dictionary but plain-Brotli rows, so nothing benefits from it yet.

Two things about the data decided the design, both checked rather than assumed:

Content over 1 MiB is not stored as independently compressed pieces. The rows
are raw 1 MiB slices of a single Brotli stream -- a slice alone does not decode
-- so the unit of work is a base row plus its continuations, concatenated,
decoded, recompressed and re-split. A naive per-row migration would have
destroyed all three such items, silently, since each slice still looks like a
blob.

And those continuation rows are numbered from -2 while WebServer's reassembly
loop starts at -1 (ADFA-5170), so they already serve truncated. The script
preserves whatever numbering it finds, keeping the migration behaviour-neutral;
--renumber-continuations rewrites from -1 instead, which makes them reachable
again, as an opt-in rather than a side effect.

Classification tries the plain decode first, deliberately: attaching no
dictionary to a stream that needs one reliably fails, so a successful plain
decode proves a row is unmigrated. The reverse is not safe -- a dictionary
attached to a stream that never used one can decode to different bytes without
erroring. A row that decodes identically both ways is left alone; those are
tiny already-compressed payloads the compressor found nothing to reference for.

Every item is verified before it is written: the recompressed bytes must decode
back to exactly the original plaintext, or the item is reported as an error and
left as it was.

Measured on a copy of the 20-Aug database, 20 workers: 29,751 items, no errors,
129.0 MiB of stored content down to 85.7 MiB (33.6%), 3.3 minutes against about
73 single-threaded. The file itself goes 313.8 MB to 267.7 MB after VACUUM, and
integrity_check passes. Verified independently of the script's own accounting:
303 sampled items, including all three chunked ones, decode with the dictionary
to content byte-identical to what the source decodes plainly. Re-running is
cheap (0.1 min) and converges -- pass two rewrote one row 11 bytes smaller,
passes three and four changed nothing.

--limit and --path did not reach the renumber phase, so a scoped trial run --
the first thing anyone sensibly tries -- rewrote every chunked item in the
database. Verified: --limit 1 --phases renumber --yes renumbered 2 of 2 items
before, 1 of 1 after.

Every run demanded a CompressionDictionary, and a database without the table
crashed with a traceback instead of the intended message. Only retype and
migrate decode, so renumber now runs without one -- which is exactly the old
database whose numbering most needs repairing -- and a missing table says so
and says which phase still works.

The phases-without-migrate path returned 0 whatever it had printed. Failures of
the work a phase exists to do (a decode that fails, a renumber that cannot
proceed, a verification mismatch) are now errors and set a non-zero exit;
observations that do not make the run wrong (left as text, a name disagreeing
with its payload, a chunked item whose base row is not exactly 1 MiB) stay
notes. Both are labelled in the output.

Worker dictionary files were never deleted: 160 of them, 40 MB, had accumulated
in /tmp from earlier runs. Each worker now unlinks its own at exit.

.bmp was nominated as a candidate but sniff() had no BMP signature, so a real
BMP was reported as carrying no binary signature -- the opposite of the truth.

The mislabel sweep used startswith("text"), the same media-type boundary bug
fixed in ADFA-5241, which calls textual/example a text type. The candidate SQL
had it too. Both match at the boundary now, via one helper that says why.

A dry run printed "renumbered from -1: N" having written nothing, and the
wording had the direction backwards. It now says "would renumber to start at
-1: N".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…readable

jatezzz, high: the script recompressed content against the dictionary and never
declared a version, so the app -- which gates on the declared MAJOR, not on the
presence of CompressionDictionary -- would refuse to attach the dictionary and
every row just migrated would fail to decode. A database that looks migrated and
serves nothing. The migrate phase now writes major 2 in the same transaction as
the last batch, and says so; a dry run says what it would declare and why.
Declared even after a partly failed run, since WebServer falls back to a plain
decode for rows that did not migrate but cannot read the ones that did without
it.

jatezzz, medium: newly inserted continuation rows carried the base row's
languageID, while WebServer looks continuations up with "languageId = 1"
hardcoded. Any item whose base row is not language 1 would have had its
continuations become invisible and its page truncate at 1 MiB -- the exact
ADFA-5171 symptom this script exists to remove. Continuations are inserted as
language 1 now.

jatezzz, medium: retyping into a type whose own compression is not 'none' left
the bytes compressed under a type they did not match, produced two spurious
verifier errors per row, and could have served raw compressed bytes. Such a
target is now refused with an actionable message and the row left alone.

jatezzz, low: one missing or NULL-content row aborted the whole run from inside a
worker, with earlier batches committed and no summary. Both phases now report it
and continue. Testing that found a second path to the same crash: NULL content
also made LENGTH() NULL, so the phase summary threw before any row was read.

jatezzz, low: phase 1 submitted every candidate at once and held every decoded
plaintext resident. It batches now, like phase 3.

CodeRabbit, major: renumber_item parked rows under "{base}-renumbering-{n}",
which a real row can already occupy -- failing on UNIQUE(path) after the
collision checks had passed. A single ascending pass needs no temporary names:
contiguity is verified first, so the lowest target is free and every later one
was vacated by the move before it. My comment claiming otherwise was wrong.

CodeRabbit, major: verify_retype reported valid -2 suffixes as errors in a
retype-only run, so a successful run exited 1. That check now runs only when
renumbering was requested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

All three follow from the previous round and are the reviewer's, not mine.

Batching phase 1's submissions bounded the workers, not the memory: every
completed Inspection was appended to a list and written only after the last
batch, so --batch did nothing about the thing that actually runs out. Each batch
is now inspected, typed, written and committed before the next one starts, and
each item's decoded payload is dropped as soon as it is written. The verifier
gets a set of paths rather than a list holding slices.

The continuation language fix only covered inserts. Reusing an existing
continuation row updated its content and type but left its languageID, so a row
that predates this script and carries the base row's language stayed invisible
to WebServer's continuation query -- the same truncation, through the row this
script chose not to replace. The update normalises it too.

The version was declared after the last batch, so a run interrupted between two
committed batches left dictionary-compressed rows in a database still declaring
a version the app will not attach the dictionary for: every committed row would
fail to decode. It is now declared in the same transaction as the first batch of
migrated content, with the end-of-run declaration kept as the fallback for a run
that migrates nothing but finds content already migrated.

Verified: a 4-row database with a realistic dictionary declares 2.0.0 during the
batch loop rather than after it; retype with --batch 1 still retypes and still
refuses a compressed target type; and the earlier round's checks all still hold
-- the parking-path squatter, the NULL-content row, language 1 on inserted
continuations, no suffix complaints in a retype-only run, and a dry run that
leaves the file byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…n grouping, guard v2 declaration

Three fixes from review:

- declare_dictionary_version no longer DELETEs the version log. The table
  is append-only by contract (docs/documentation-database.md,
  DatabaseVersionResolver): the last-inserted row wins, so the INSERT
  alone declares version 2 and prior rows stay as history. Docstrings
  that asserted a one-row contract are corrected.

- load_items only groups a "-<digits>" sibling as a continuation when
  its base row holds exactly CHUNK_BYTES, the app's own chunk-detection
  rule. Grouping on the name alone let phase 1's rewrite of a short base
  absorb an independent sibling page's bytes and delete its row. Every
  phase inherits the gate from this one choke point; phase_renumber's
  local copy of the test is now redundant and reduced to a comment.

- Declaring version 2 over rows left plain is guarded two ways:
  --only-if-smaller is refused whenever the migrate phase runs (it
  deliberately leaves plain rows in a database that will declare 2, and
  a plain row can decode against the dictionary to wrong bytes without
  erroring), and a write run that declared 2 ends with an explicit
  WARNING when any brotli item did not migrate.

The guard I added last round -- base row exactly CHUNK_BYTES -- rules out
a small base and nothing else. A real page that happens to be exactly
1 MiB, sitting next to independently named "-2"/"-3" pages, was still
grouped with them, and phase 2 renamed those pages into its slice slots.
Reproduced against the real schema: p/page (1,048,576 bytes) plus
p/page-2 and p/page-3 at 11 bytes each came out as p/page, p/page-1
(11 bytes), p/page-2 (11 bytes) -- both original URLs 404, one page
gone, and the app appends a foreign page's bytes when it reassembles.
The comment asserting this could not happen was wrong.

The signal was already loaded: a genuine slice set has every slice
except the last at exactly CHUNK_BYTES, because that is how the writer
splits. An 11-byte "-2" followed by a "-3" is provably not one. The test
now runs in load_items, so all three phases inherit it rather than phase
2 alone, and a sibling set that fails it becomes independent items
instead of being silently absorbed. Verified: the two coincidence cases
are left untouched, and both genuine mis-numbered slice sets are still
repaired.

A continuation whose base never became an Item was dropped silently --
not migrated, not counted, not reported, in a database the run then
declares version 2. It is reported now.

The version declaration is refused on a --path or --limit run. It covers
the whole database, so only a run that considered the whole database may
make it; a scoped run left tens of thousands of rows plain while telling
the app they were dictionary-compressed. Most such rows throw and fall
back, but a fraction decode without error to different bytes, which the
code's own comment says two lines further down. The run now prints why
it withheld the declaration.

Phase 3 releases result.slices after writing, which phase 1 already did
with a comment explaining why.

Found in review of PR #1724.

…lags

Three findings from CodeRabbit's review of PR #1724, each verified against
a scratch database before and after:

- Guard future.result() in the retype and migrate as_completed loops. A
  worker exception (e.g. the brotli CLI vanishing mid-run) re-raised and
  aborted the phase with earlier batches already committed and no summary;
  it is now recorded as that item's error and the run completes. A
  shutil.which("brotli") preflight also refuses retype/migrate runs
  up front when the CLI is missing, next to the dictionary checks.

- Validate --batch and --workers >= 1. Zero raised from range() or
  ProcessPoolExecutor after work may have started; a negative batch
  silently processed nothing. Both now exit 2 with a clear error,
  alongside the existing phase-name check.

- verify_retype passed read_blobs() straight to b"".join(), so a row
  deleted or NULLed between the write and the verify aborted verification
  with a TypeError. It now records the problem and continues, matching
  the adjacent vanished-row branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197g8vkUQ1d6oLNbi8EnAYe

…uageID

Three fixes in load_items/renumber_item:

- A row split back out of a disproved continuation group was built from the
  base item's languageID/contentTypeID/templateId, so phase 3 relabelled an
  independent page with the base's MIME type. It now carries its own row's
  metadata.
- Orphaned "-N" rows (whose would-be base is itself a continuation) were
  logged and dropped, so a plain-brotli row silently survived a run that
  declares version 2. They now migrate as standalone items; a genuine stray
  slice fails to decode and feeds the existing "did not migrate" warning
  instead of vanishing.
- renumber_item's UPDATE moved only the path. WebServer loads continuations
  with "languageId = 1" hardcoded, so a renumbered row under another language
  stayed invisible and the page still truncated. The UPDATE now normalises
  languageID like write_item does.

Verified on scratch databases: split rows keep type/language/template through
a migrate run; a migratable orphan round-trips against the dictionary while an
undecodable one exits 1 with the version warning; renumbered continuations end
at languageID 1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197g8vkUQ1d6oLNbi8EnAYe

…A-5153-dictionary-migration-script